Leveraging recent advances in deep learning for audio-Visual emotion recognition

Recent Advances in the Automatic Recognition of Audio-Visual Speech

Visual speech information from the speaker’s mouth region has been successfully shown to improve noise robustness of automatic speech recognizers, thus promising to extend their usability in the human computer interface. In this paper, we review the main components of audio-visual automatic speech recognition and present novel contributions in two main areas: First, the visual front end design,...

Recent Advances in EMG Pattern Recognition for Prosthetic Control

Audio-Visual Spontaneous Emotion Recognition

Automatic multimodal recognition of spontaneous emotional expressions is a largely unexplored and challenging problem. In this paper, we explore audio-visual emotion recognition in a realistic human conversation setting—the Adult Attachment Interview (AAI). Based on the assumption that facial expression and vocal expression are at the same coarse affective states, positive and negative emotion ...

Multimodal Transfer Deep Learning with Applications in Audio-Visual Recognition

We propose a transfer deep learning (TDL) framework that can transfer the knowledge obtained from a single-modal neural network to a network with a different modality. Specifically, we show that we can leverage speech data to fine-tune the network trained for video recognition, given an initial set of audio-video parallel dataset within the same semantics. Our approach first learns the analogyp...

Noise Analysis in Audio-Visual Emotion Recognition

This paper describes the use of a decision-based fusion framework to infer emotion from audiovisual feeds, and investigates the effect of noise on the fusion system. Facial expression features are constructed from linear binary patterns, and are processed independently of the prosodic features. A linear support vector machine is used for the fusion of the two channels. The results show that the...